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This paper seeks to provide a rationale for 
evaluating Technical and Further Education (TAFE) programs by using a 
management system approach that is based on corporate planning. The 
first section reviews the sources of increased demands for 
accountability in TAFE (societal, economic, government, and 
legislative sources) and examines various management systems that 
could be used to meet increased demands for accountability (program 
budgeting, program evaluation, management information systems, 
management procedures, and corporate planning). The first section 
concludes with an examination of the Swedish model for 
self-evaluation. The second part of the report describes the 
following aspects of the attempt in New South Wales, Australia, to 
increase accountability by using a corporate nlanning-based approach 
in which evaluation is a major component in providing accountability 
and maintaining quality of educational provision. The following 
topics are covered: the diversity and scope of TAFE in New South 
Wales, corporate planning as a management tool, strategic planning 
(departmental, college/school, and servi':ing unit strategic plans), 
staff participation in strategic planning, action or operational 
planning, evaluation and monitoring, performance indicators and their 
interpretation, evaluation procedures for program monitoring, 
comprehensive audits and program evaluation activities, establishment 
of a directorate of audit and review, program performance evaluation 
plans, planning and e/aluation support to managers, and future 
directions. (MN) 
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FOREWORD 



Senior Fellowships enable senior staff from TAFE Authorities to 
spend 6-12 months at the Centre to undertake studies of 
national significance. The Fellowships are generously supported 
by all TAFE Directors, 

The first Senior Fellow was Mr John White, Deputy Director 
General (Planning and Review) of the NSW Department of TAFE, 
His Fellowship was especially appropriate for three main 
reasons. First, it would have been impossible for the Centre to 
have obtained a more senior person and a more suitable person to 
be its first Senior Fellow, 

Second, the actual idea of the Senior Fellowships arose out of 
early discussions with Dr Allan Pattison, the Director General 
of the NSW Department of TAFE, and he showed his very practical 
support for the idea by releasing Mr John White, 

Third, the nature of the topic chosen for the research, 
educational accountability, is of crucial importance, ' Thus this 
paper will be eagerly read throughout all of Australia's TAFE 
Authorities. 



William C Hall 
Adelaide, June 1987 
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RECOMMENDATIONS 



While this project is in the format of an investigative report, 
the nature and extent of recent developments in NSW TAFE, 
particularly those experiences relating to the - 

(a) restructuring of senior management 

(b) the introduction of the corporate planning 

(c) decentralisation and regionalisation of the department 
administration 

(d) establishment of an Audit and Review Directorate 

(e) introduction of program evaluation 

provide an appropriate basis for a number of recommendations on 
courses of action which may be of interest to other TAFE 
departments. 

Based on the NSW TAFE experience it is recommended that in 
establishing procedures to meet accountability and planning 
requirements, consideration be given to: 

1. the introduction of a corporate planning management 
process. This process should include the development of a 
corporate plan detailing corporate and strategic goals, and 
the development of strategic and action plans, and program 
evaluation; 

2. establishment of an appropriate data base using generally 
agreed upon performance indicators; 

3. utilisation of established assessment procedures where 
available to avoid unnecessary duplication of effort in the 
evaluation phase; 

4. establishment of a separate Audit section which draws 
together existing financial auditing functions and extends 
these to cover requirements relating to effectiveness, 
efficiency and appropriateness of TAFE operations; 
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5. use of institutional evaluation as a final stage in the 
evaluation procedure. Institutional evaluation should be 
used only where ongoing evaluation indicates the need for a 
more thorough review of college activities; 

6. development of college profiles or areas of activities. 
This may be required as a forerunner to the preparation of 
a strategic plan or as part of the evaluation of a college. 
The profile should identify the aims of the institution, 
the nature and extent of the TAFE clientele it serves, and 
the range of resources available to achieve the stated 
goals; 

7. encouragement of staff involvement in the development of 
profiles, strategic and action planning and program 
evaluation. It is important for all staff to question the 
effectiveness and efficiency of their operations; 

8. involvement of all managerial staff in staff development 
activities relating to corporate planning and program 
evaluation; 

9. inclusion of a computer systems review unit as part of a 
comprehensive audit section; 

10. publishing of outcomes of evaluations to provide an 
appropriate measure of accountability. This may be through 
the provision of annual reports or other publications in 
which the selected data is analysed and future directions 
identified. 
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Educational accountability in TAFE is not a new issue. It 
has, however, assumed greater importance in recent times. 
This paper examines the reasons for this new emphasis. To 
meet this challenge, TAFE in Australia needs to review its 
organisational structure and management system in order to 
provide the degree of flexibility and effectiveness required 
by government initiatives and changing community needs. 
Similar events are occurring in other Western countries and 
some of these overseas developments ar^. t^xplored. 

This paper suggests a management system approach involving 
corporate planning of which evaluation is a major component 
in providing accountability and maintaining quality of 
educational provision . 
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SUMMARY 



This paper seeks to provide a rationale for the extension 
and reinforcement of an educational management process based 
on corporate pxanning. Such a development is required in 
order to accommodate increasing demands for accountability, 
and the need to be flexible and innovative in operation in 
providing for government initiatives and the changing needs 
of society. The first part of this paper reviews the 
sources of these demands for increased accountability in 
TAFE while the second part describes one state TAFE 
department ' s response to these challenges . 

To provide for these requirements, as well as to improve the 
level of decision-making in TAFE, a corporate planning model 
including internal or* self-evaluation is advocated. 

Present procedures for institutional evaluation are reviewed 
to establish their appropriateness to TAFE requirements. 
The models for institutional evaluation as applied to higher 
education in the USA and other Western countries, in which 
each institution is evaluated as a separate entity, are seen 
to be of limited value to present requirements in TAFE . 
Such evaluations are costly, provide information 
infrequently and adversely affect staff morale. 
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The activity-based "self-evaluation" model operating in 
Sweden is seen as more appropriate to TAFE needs. This 
model is compatible with the corporate planning process 
introduced in NSW TAFE in 1983. 

TAFE systems in other Australian states and territories 
differ in some respects from the NSW system, but there are 
also similarities of goals, administrative procedures and 
educational provision. These similarities, together with 
commonalities in government intent, provide justification 
for a similar restructuring of their educational management 
systems. Recent experience within NSW TAFE with regard to 
corporate planning is used to support this contention. 

This paper proposes that evaluation of TAFE's educational 
activities is essential both to meet the demands for 
increased accountability from all stakeholders in TAFE, and 
to provide a proper basis for planning. 
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PART I 



EDUCATIONAL ACCOUNTABILITY IN TAPE 

Introduction 

Accountability - The New Dimension in TAPE 

The argument for increased accountability in education has 
reached new dimensions in recent times with the realities of 
limitations in funding and the demands of stakeholders for 
more effective and efficient utilisation of scarce 
resources. These developments have broadened the impact of 
accountability from simply financial accounting to 
justification for action taken. This need for justification 
holds the education authority accountable for policy and 
decision-making and brings into focus the question of 
quality of the educational provision. This in turn raises 
questions as to the appropriateness of the educational 
programs r the availability and utilisation of physical 
resources, the nature of the learning environment and the 
teaching methodology. Thus, educational accountability has 
the potential to provide a window through which stakeholders 
nu\y view the educational process to judge its effectiveness, 
efficiency and the quality of its educational outcomes. 
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Cor.!nitment to the concept of accountability therefore 
presents a major challenge to educational managers. 

In this context, Bartlett (1971) defines jountability as 
the effective and efficient attainment of worthwhile 

educational outcomes that are reflected in student 
behaviour'. This emphasis on student outcomes directs the 
requirement for accountability on all those involved in the 
educational provision. Norton (1972) refers to this concept 
of shared responsibility of all educational staff as joint 
accountability. He suggests that legislation can be used to 
strengthen this responsibility to be accountable as can 
pressure from students, employers and the community in 
seeking to promote the moral and social obligations of the 
education system. To this concept must also be added a 
professional obligation which is influenced by the academic 
environment . 

The means by which the window on the educational process can 
be opened to stakeholders is through the availability of 
accurate information on programs, procedures and outcomes. 
Terrell & Bass (1979) suggest that information communicated 
in this way must be presented so that it generates support 
for action taken. This introduces the concept of promotion 
of an acceptable image to generate confidence, and hence 
support for the educational provision. 
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The provision of factual information to stakeholders for 
accountability purposes is reinforced by Wells (1986) who 
states that educational accountability is the 
responsibility or obligation of all staff of an educational 
department or institution for the accurate reporting of 
objectives^ strategies^ inputs^ ou*;puts and the impact of 
the educational provision. It involves making such 
reporting open to view by keeping records about decisions/ 
activities and results and communicating these to higher 
authorities^ to staff and clients and the community in 
general ' • 

This interpretation of accountability is applicable to TAFE 
particularly when the nature and extent of the educational 
provision/ the requirements of students and the needs of 
industry and commerce are taken into consideration,. 

This demand for increased accountability should be viewed 
positively and should not be taken as implying any form of 
coercion, Terrel & Bass (1979) stress the positive aspects 
of accountability which/ in their view/ can raise the 
respect for, and understanding of, the educational 
institution • The need for accountability also provides for 
a more co-operative approach to education and training in 
that it recognises the contributions and responsibilities of 
all stakeholders to the total educational process. 



Nevertheless, the need for care must be exercised in the 
gathering and reporting of the data to be used for 
accountability purposes. The view through the window can be 
distorted and to explain such occurrences McNelly and 
Cameron (1981) have introduced the concept of 'fallacious 
accountability'. They suggest that the distortions which 
lead to fallacious accountability are usually due to the 
adoption of incorrect or inappropriate measures of the input 
to and impact of the educational programs and that closer 
links with stake-holders to promote agreement of acceptable 
measures of performance are needed. Without these agreed 
measures, they suggest validity could be lacking and 
misinterpretation can result. As illustration of this 
possible misinterpretation, the NSW Department of TAFE 
offers a range of one-year fulltime pre-apprenticeship 
courses. Some graduates of these courses seek and obtain 
apprenticeships or employment in related vocational fields 
while others continue their studies either in TAFE or at 
other tertiary institutions. The department considers such 
post-course activities as appropriate indicators of program 
fulfilment. The view of some external stakeholders, 
however, restricts the measure of success to the obtaining 
of an apprenticeship, thus leading to a lack of agreement on 
the value of such programs to meet the needs of students. 
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other concerns relating to the provision of accountability 
are : 

(i) the extensive time, energy and resources which may 
need to be expended on establishing and maintaining an 
accurate data base upon which accountability is to be 
provided. (This raises questions about the efficiency 
of the accountability mechanism) . 

(ii) the degree to which the stakeholders use the 
information provided. (This relates to the 
effectiveness of the process) . 

These concerns do not detract from the essential concept of 
gathering relevant information for accountability purposes. 
It is important, however, that the information gathered be 
in usable form, and that it be put to appropriate use. 

The Value of a Data Base 

Any data base used for accountability purposes is acquired 
through the evaluation of the educational input and 
outcomes. This evaluation process also has an 

administrative purpose, the importance of which is stressed 
by Norton (1972) who refers to evaluation as the 
process of collecting and providing useful information for 
the purpose of making sound decisions'. Smith (1985) also 




refers to the planning Implications of the evaluation 
process by emphasising the need to measure inputs and 
outcomes which will: 

(i) assist in planning, policy development, implementation 
and monitoring of programs in operation at all levels 
of management; 

(ii) provide a measure of appropriateness of organisational 
goals and objectives and the need for improvements in 
performance ; 

(iii) satisfy accountability requirements to stakeholders 
including students, the community, the educational 
authority and government • 

The evaluation process is considered to be comprehensive in 
scope because of the broad range of uses for which the data 
base is intended. 

Thus, comprehensive evaluation provides an information data 
base for future decision-making with regard to the education 
provision, and for accountability purposes where accuracy of 
reporting and documentation of the information are crucial 
elements and emphasis is placed upon the justification of 
decisions made and actions taken, (Figure 1) . 
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The Role of Comprehensive Auditing 

Evaluation has been defined as an integral part of the total 
management process and completes the planning operations 
cycle. The role of auditing as an official examination or 
review is separate from the management process. Previously 
auditing was mainly concerned with the verification of 
financial accounts and records but consistent with this move 
towards increased accountability there has been a broadening 
of the scope of auditing from purely financial 
considerations to a more comprehensive approach. This 
expansion in role places greater emphasis upon the 
efficiency and effectiveness of the use of resources in 
achieving educational goals. Comprehensive auditing reviews 
the criteria and standards upon which performance is based 
and examines the relationship between program outcomes and 
resource inputs to determine the level to which the 
organisation is managing its resources and the degree to 
which its performance is maximised. 

Chelmsey (1985) believes that the auditing function needs to 
be external to the organisation, but this is not always 
essential or in some instances even desirable, unless there 
is a real need to compare one organisation with another. In 
all cases, however, auditing differs from comprehensive 
evaluation in that it is an independent procedure and should 
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be conducted by a body separate from the field of 
operations. 

The report of the NSW Public \ccounts Committee (1985) 
states that objective setting and performance review are 
functions of the organisation and accordingly recommends 
that each department/authority be required to continually 
review the performance of its organisational units and 
programs. The committee suggests that such audit reviews 
can give support to management in that they provide 
educational staff with encouragement, support and 

constructive suggestions for improving the effectiveness of 
programs ' • 

Figure 1 d iagrammatically relates the auditing or review 
process to the management process. 

Sources of Demand for Accountability 

Demand for increased educational accountability has come 
from various quarters including socie*:al/ the economy and 
legislation, 

(a) Societal Influences ; 

Two major social factors influencing the trend to 
greater educational accoun tabili tv are firstly, 
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increasing public awareness of, and commitment to, a 
range of human rights issues and, secondly, the 
expanding role of government in provision of community 
support services. 

These influences are apparent in the expectation by 
both the government and the public that TAFE should 
provide a broad range of educational packages to meet 
the emerging needs of disadvantaged groups in the 
community (including unemployed youth. Aborigines and 
the disabled) . 

At the same time, TAFE is eijpected to continue to 
provide industry and commerce with an adequately 
trained workforce, and also to provide a range of 
personal enrichment programs for those seeking 
improvements in the quality of their lifestyles. 

TAFE has responded to these expectations with 
flexibility and innovativeness . This responsiveness, 
and the recent prominence given to TAFE by government 
and media, have resulted in greater public awareness 
of TAFE's goals and objectives; increased public 
acceptance of TAFE as a major provider of tertiary 
education; wider interest in the role and function of 
TAFE in the community; and a concomitant demand for 



ERIC 



23 

10 



accountability in relation to the provision of 
educational facilities and availability of programs* 

In the current atmosphere of fiscal restraint there is 
recognition of the limitations to government spending 
and a concern for the effectiveness of government 
initiatives : equity of access to these finite 
resources is increasingly seen as a significant 
element in the measurement of effectiveness. 

The increased participation of stakeholders in the 
decision-making process of TAFE, the commitment on a 
national scale to common core curricula, course 
registration and standard nomenclature are further 
developments which enhance this accountability 
process. 

Economic Influence 

The current economic climate with its unacceptable 
levels of youth unemployment , its need for industry 
re-structuring and its preparation for the 
introduction of new technology has led also to 
increased demands upon TAFE to provide a range of 
educational programs for the training and re-training 
of the workforce , increasing the competitiveness of 




the unemployed for eniployment opportunities and 
imparting skills necessary for the full utilisation of 
new technology. These increased demands brought about 
by the state of the economy and the restructuring of 
industry are in conflict with the need for financial 
constraint applied to public monies. This has 
resulted in moves at both commonwealth and state 
levels to improve government administration generally 
and to generate an information flow upon which the 
allocation of future resources can be provided with 
increased precision. This has led to i commitment by 
governments to a system of accounting which classifies 
costs according to outputs instead of inputs. This 
concept is known as program budgeting and is affecting 
all government departments and statutory authorities. 
It is designed to provide betteri insight into public 
spending and to encourage expenditure by objectives, 

(c) Government Initiatives and Legislation in the 
Public Sector 

If government initiatives and legislation relating to 
the public sector are examined, the trend towards 
increased accountability is evident. 

At the state level, and again using NSW as an example, 
government initiatives directed to all departments, 

25 
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including TAFE, reveal a pronounced trend towards the 
provision of information for accountability purposes • 
This accountability is required not only in the 
financial sense but administratively as well, as it is 
designed to ensure that government departments and 
statutory authorities are providing the services for 
which they were established. 

The beginning of this trend for increased 
accountability can be traced back to the commission 
established in 1976 to report on government 
administration in NSW which was headed by Professor 
Peter Wilenski (1977). As a result of the 

recommendations of this reports there have been a 
succession of statutory and other initiatives designed 
to make governments, statutojry authorities and 
agencies more accountable for their operations. 

These initiatives include: 

(i) The Public Service Act (1979) , which 
specifically makes the ministers and 
departmental heacs responsible for the 
effectiveness and efficiency of their 
department's activities. The Act also provides 
the statutory basis fo:: external efficiency 
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audits to be undertaken at the request of the 
Premier in government departments and agencies; 

(ii) The Public Finance and Audit Act (1983), which 
requires departmental heads to ensure that there 
is an effective system of control and an 
internal audit procedure in their departments 
which reviews programs to establish consistency 
of stated goals with operations; 

(iii) Management strategy reviews of selected 
departments and agencies which are required ♦•.o 
be undertaken by the Premier's Department. 
These reviews provide an external investigation 
of key questions concerning the trends and 
purposes of the organisation. The reviews 
include an assessment of the degree of 
congruence between government policies and 
initiatives and organisational aims and 
objectives ; 

(iv) The introduction of Program Budgeting by the NSW 
State Treasury which is designed to forge a more 
direct link between a department's budgetary 
allocation and its programs of expenditure. 
Program budgeting allows i >i the costs of 
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programs (sets of activiti es) to be known and 
requires that program effectiveness be assessed 
and taken into account in future resource 
allocations ; 

(v) Moves towards a closer scrutiny of departments 
by the NSW Public Accounts Committee (PAC) • All 
government departments are required to complete 
a detailed questionnaire on their operations and 
the management procedures used to evaluatv^ the 
efficiency and effectiveness of those 
operations; 

(vi) A growing accountability to the community 
through the Ombudsman's Office which 
investigates complaints received from members of 
the community relating to services provided by 
government departments and agencies; 

(vii) The annual reporting of departmental performance 
as required by the Annual Reports (Departments) 
Act of 1985. These reports must include 
qualitative and quantitative measures of 
performance to show the level of efficiency and 
effectiveness / details of consumer or client 
reaction to the services provided and an account 




of the major issues and problems encountered 
during the year of reporting; 

{viii)The requirement that departments conduct self- 
examination of their programs (known as internal 
program performance reviews) using established 
program evaluation methodology. The rolling 
5-year plan for future evaluations is to be 
submitted annually to the Management Performance 
Committee, and the annual report must include a 
description of the nature and extent of the 
reviews undertaken, and the improvements to 
performance which have resulted from these 
reviews . 

A diagram of these measures of account *ibility as 
implemented in NSW is shown in Figure 2* 
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Government Influences R elating to TAFE 

If the situation with regard to government influences 
relating to TAFE is examined at the state and national 
level this general trend of accountability is very 
evident. 

The Kangan Report (1974) described TAFE's existing 
role and function and projected TAFE's capacity to 
improve access to tertiary education of disadvantaged 
groups in society. The report provided a new standard 
for TAF£'s involvement against which its activities 
could be evaluated. The report prompted the 
introduction of a new TAFE Act which gave rise to a 
more descriptive title for this field of tertiary 
education and established legislative requirements for 
the future. The Williams Committee Report (1979) also 
emphasised the need to evaluate the post-secondary 
educational process. 

Each state has also undertaken at least one 
comprehensive review of its TAFE activities during 
this period, with similar intent and uniformity in the 
conclusions reached . 

Although the Act setting up the Tertiary Education 
Commission (TEC) and its councils makes no direct 
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reference to accountability issues^ the need for 
consultationw with the states was included. The 
States Grant (Tertiary Education Assistance) Act, 
however^ does provide for the states to supply such 
information as che TEAC may require. 

Requirements with regard to Special Purpose Grants and 
funding provided for Pre-Apprenticeship Programs^ 
Labour Adjustment Training Arrangement (LATA) and 
Priority 1 Traineeships have more specific 
accountability requirements as a conditi *^ of funding* 

Clearly then^ the concept of accountability is 
extending; far beyond mere financial transactions and 
procedures and is moving towards increasing the 
evaluation and review of the efficiency^ effectiveness 
and even appropriateness of a department's functions 
and operations. With this growth in awareness of 
accountability to stakeholders in TAFE/ it falls 
within the bounds of possibility that instances of 
legal recourse, similar to those occurring in the USA 
relating to professional negligence, may occur. 

In fact, the forces tending towards greater 
educational accountability in Australia have marked 
parallels in other western countries. 
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Educational Accountability in other Countries - Are we 
Treading the same path? 

An examination of trends in educational accountability in 
other western countries reveals marked similarities with 
recent developments in Australia, 

In comprehensive surveys of trends in higher education 
management in five European countries (Lockwood & Prosser. 
1980; Broadfoot, 1982) a number of key areas for comparison 
were identified by the researchers. These include: 

(i) changes in educational needs and clientele; 

(ii) new legal requirements and restrictions; 
(ii\) the effects of economic uncertainty; 

(iv) organisational complexity due to increased demand and 
rapid growth; 

(v) industrial relations; 

(vi) demands for increased productivity and public 
accountability. 

Responses to these key issues from each of the countries 
surveyed - Ireland, UK, France, Sweden and Denmark - were 
similar, although to varying degrees. Changes in needs and 
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student profiles were apparent, but in contrast with USA and 
Australia, there was less response to claims by minority 
groups. In fact, disadvantaged groups were not even 
identified as a significant feature of the surveys. 

In all countries, with the exception of Ireland, national 
legi.?lation was seen as an increasingly pervasive influence 
on the demand for greater educational accountability. 
Legislation relating to issues such as occupational health 
and safety, sexual equality, privacy and confidentiality 
were taken as indicators of the impact of recent government 
initiatives which now affect all sectors of society in these 
countries. Increases in the use of litigation to resoJLve 
issues were also in evidence. 

The survey identified the growth of tertiary education and 
the state of national economies as important influences on 
the demand for increased public accountability. Increased 
demands for information and tighter controls on the transfer 
of funds between budget items were other indicators of this 
increased level of accountability. 

The institutions surveyed did express their concern about 
increases in the level of government control and the 
diverting of funds to provide for the machinery of 
accountability. In the UK in particular, accountability 
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through the increased provision of information to the public 
usually through the publication of annual reports and other 
documents, was seen as a costly, though necessary, measure 
of accountability. 

In those European countries surveyed, economic uncertainty 
is related to the state of national economies, inflation 
rates, political aspirations and the changing status of 
higher education. This has resulted in institutions 
broadening their educational provision to meet government 
initiatives and reviewing programs to achieve economies of 
scale because of increasing financial restraint. 

Certainly greater effort has gone into effecting economies 
of scale in the educational provision. However, in the UK, 
the move to create polytechnics is seen as a means of 
increasing educational productivity at a unit cost 
substantially lower than that of universities. It is 
interesting to compare this development with the 
introduction of Colleges of Advanced Education in Australia 
in the 1960s, and the recognition and acceptance of TAE'E in 
the 1970s and 1980s. 

All countries surveyed acknowledge increases in the 
administrative complexity of tertiary institutions. Sweden 
in particular has encountered difficulties associated with 
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the organisational reforms of higher education in 1977. 
These reforms introduced a form of goal-oriented program 
budgeting and as a result higher education institutions have 
increased the complexity of their organisational and 
management structures. These problems of complexity are 
exacerbated when confronted with increasing financial 
constraints and, together with the move towards 
participatory luanagement, have led to a re-allocation of 
human and financial resources. 

Many of the developments cited in the survey are similar to 
those taking place in Australia. However ^ the researchers 
claim that the resultant increases in educational and 
administrative complexities of tertiary institutions should 
not be taken as an excuse to defer organisational change. 
In fact, these developments should stimulate action to 
provide increased flexibility in educational management. To 
achieve this flexibility the authors of the survey suggest: 

(i) cre?.ting interdisciplinary academic units; 

(ii) improving the quality and quantity of staff 
development ; 

( ii i ) increasing the lev^ . of co-operation with external 
stakeholders ; 
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(iv) reducing the complexity of the decision making 
process ; 

(V) increasing flexibility in control over budgets. 

In predicting trends and the effect they will have on 
tertiary education in these European countries, Lockwood and 
Prosser predict that there will be: 

(i) an increasing scarcity of funcs for education; 

(ii) tighter government controls; 

(iii) more demands for broader public accountability; 

(iv) the development of technology to liberate human 
resources ; 

(V) changes in educational needs due to increases in 
leisure . 

The similarities between developments in these five Eui-opean 
countries and Australia are real and predictable, as the 
educational systems in this country have been modelled on 
those in the UK. As with tertiary education in the European 
countries surveyed, it is equally necessary for tertiary 
education in Australia and particularly TAPE to prepare to 
meet the challenge of changing public and government 
expectations of TAPE and demands for increased public 
accountability of its educational provision. 
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The Need for Systematic Planning and Evaluation 

From this review of trends in educational accountability in 
other countries, it will be seen that the current trends 
towards increased accountability in this country may require 
significant changes to our present organisational structures 
and the introduction of more appropriate management systems. 

The literature on educational accountability consistently 
refers to the unstructured nature of planning and evaluation 
in education. For example. Galloway and Knell (1974) argue 
that greater accountability is achievable through syst ^tic 
and more formalised planning. Bacon (1978) believes that 
the unsatisfactory nature of present planning systems is due 
t'j the inadequacy of available information upon which 
decision making is based. This view is supported by Turner 
(1980) who attributes the recent accelerated development of 
more systematic administrative procedures used by colleges 
to the use of management information systems and variations 
of management by objectives. Schure (1971) is of the 
opinion that the introduction of sophisticated forms of 
information processing should result in more efficient 
decision making and more effective workforce planning 
procedures . 

Closer to home, the call for more effective planning in TAFE 
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has come from TAFE educational administrators. The NSW 
Report of the Joint Workshop of Regional Directors, 
Metropolitan Principals and Heads of Schools in 1981 
recommended that much more effective planning needs to 

be undertaken to ensure a more adequate response is made to 
the overall needs of and demands made upon TAFE ' • 

In considering the formal structure of education Bidwell 
(1985) refers in his article to the lack of systematic 
planning and evaluation as 'structural looseness' which he 
sees as a product of the educational process and teacher 
professionalism. Moves to change organxsat ional structure 
by decentralising the administrative process, as in NSW 
TAFE, or by imposing a greater degree of centralisation on a 
highly autonomous system, as in Victoria, are viewed as 
attempts to provide organisational structures with optimum 
fle>ribility in the educational provision. These 
organisational changes can establish also the basis for an 
improved system of information management within the 
organisation. 

Management Systems to Meet the New Challenges 

To meet the requirements of government and stakeholders for 
increased accountability and to ensure mor^ effective and 
efficient educational management, organisational 



39 

26 



restructuring and new management systems are being 
introduced into TAFE departments. Some of these changes 
result from government initiatives such as program budgeting 
while others such as corporate planning aim to provide 
improved decision-making, 

{ a ) P cogram Budgeting 

The need for more systematic planning in education 
coupled with improved systems for the collection and 
dissemination of information is reflected in the 
commitment by state governments in Australia to 
program budgeting and program evaluation. It is 
generally agreed that traditional budgeting processes 
provide control of expenditure of public monies but 
make limited provision for planning and decision 
making. Program budgeting provides a more appropriate 
information base upon which decision making at both 
government and organisational levels can be based. 

In NSW, the Review of NSW Government Administration 
(1977) recommended the adoption of program budgeting 
in order to facilitate budgetary choice by 
Ministers among departmental activities to permit the 
allocation of resources by departmental managers to 
achieve the best results and to increase public and 
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parliamentary understanding of government programs ' • 
The references to facilitating public and 

parliamentary understanding' and the allocation of 
resources to achieve the best results are very clear 
refere^.ces to the need for increased accountability of 
government departments and agencies* By 1987/ all 
governmenu Jepartments in NSW will be required to make 
forward estimates on a program basis* 

Other Australian states have given consideration to 
program budgeting, with South Australia and Victori- 
well advanced in its implementation. 

Program Evaluation 

The attention given to program budgeting as a means of 
improved decision making has led to an increased 
emphasis on program evaluation as a form of 
comprehensive audit. The NSW Public Service Board's 
Program Evaluation Unit (PEU) provides assistance to 
government departments and authorities to establish 
procedures and guidelines for the review of program 
efficiency and effectiveness, and the inclusion of 
program evaluation in the planning and budgeting 
cycle. This unit is responsible to a committee of 
senior officers from the Premier's Department, Public 
Service Board and Treasury. 
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(c) 



Management Information Sy:rftems 



These developments, together with the minimum 
standards for annual reporting now established by 
legislation, require departments including NSW TAFE to 
proviae set information for accountability purposes • 
In order to achieve this , appropriate management 
information system (MIS) is required in TAFE. Such a 
system will not only provide for accountability 
requirements of the public and government, but is 
essential also for internal decision making to achieve 
the maximum and most effective use of resources. 

(d) Management Procedures 

Linked with the need to review the structure of the 
educational c janisation and the introduction of an 
appropriate management information system, is the need 
to introduce management procedures whi ch will 
accommodate the planning for and evaluation of the 
educational outcome. In seeking the appropriate 
management procedures, educational administrators have 
adopted industrial management techniques with varying 
degrees of success. Polczynski and Thompson (1980) 
state that while a high level of success is reported 
for Management by Objective (MBO) in industry, its 
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application to higher education has proved less 
successful. A high level of participatory management 
and personal commitment to the process is required for 
its successful operation. This view is supported by 
Furamak (1981) and by Taylor (1975) who, after 
attempting to introduce MBO into schools in the UK, 
reported it as a 'valueless exercise'. Williams 
(1?82) claims that a re-thinking of the value of MBO 
to educational institutions has occurred recently and 
he reports success in the introduction of MBO to a 
large college of further education in the UK. Here 
the success is based upon the involvement and 
commitment of all staff and the appropriate 
modification of management style and organisational 
structure. 

Corporate Planning 

Some of the more successful features of MBO - goal 
setting, appraisal and analysis of results, and 
improvements in communication - arc incorporated in 
the concept of corporate or strategic planning. 
Corporate planning is a management process based upon 
the clear identification of goals and objectives. 
Thic is achieved by taking into account the perceived 
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roles and functions of the orcanisat ion , the operating 
environment and climate, and the needs of 
stakeholders. The goals so identified provide 
direction for the development of action plans which 
translate the goals into operational programs which 
require evaluation on an on-going basis. 

Analysis of the comparison of operational outcomes 
with goals provides the essential feedback upon which 
future decision making is based. Any mismatch which 
is identified signals the need to review goals aud/or 
action plans or may identify changes to the 
environment within which the organisation functions, 
and chus, may require modifications to the statement 
of organisational goals and objectives. 

The total corporate planning process provides the 
potential for improved communication, pc; ^cipatory 
management and awareness of and commitment to the 
quality of product or service. 

Evaluation - the Forgotten Factor? 

The emphasis on corporate goals and objectives, the need to 
develop profiles and appropriate action plans and the 
importance of evaluating outcomes and impacts of the 
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In reviewing the evaluation of Australian universities, 
Clarke and Birt (1982) regard institutional evaluation as a 
change agent which assists universities to adapt to changing 
environments, Lxndsay (1981) argues for evaluation to 
provide information which is relevant to management needs. 
In supporting an institutional approach to evaluation which 
will avoid fragmentation of information obtained, he 
stresses the need for the relevance of analysis to the 
decision making process. In doing so, he provides support 
for the concept of program evaluation. 

Dressel (1976) defines institutional self-evaluation as 
a planned and organised enquiry by the staff of the 
institution into the total effectiveness of institutional 
operations*. Such a definition implies a regularly spaced 
evaluation wherean for on-going management purposes, a 
continuing monitoring procedure has greater value for future 
decision making. 

Because of difficulties in developing appropriate measures 
in the form of performance indicators , reliance has been 
placed upon the professional judgement of evaluators to 
measure effectiveness (Lindsay, 1981) . Evaluation of higher 
educational institutions using professional judgement only 
and involving staff from other institutions is not viewed as 
a useful substitute for valid evaluation criteria or 
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performance indicators. In addition to this limitation, 
Lindsay argues that the major disadvantages of this approach 
to evaluation are that it is time consuming, 
administratively disruptive, costly to undertake and 
provides information for decisional purposes at infrequent 
intervals. 

Where institutions are homogeneous with respect to goals and 
educational provision, institutional comparisons can provide 
data on their relative effectiveness and efficiency. It 
could in fact be argued that such similarities of purpose 
and process occur in TAFE colleges, but such an assumption 
does not take into account the diversity of educational 
output and the variations in the needs of the communities 
served by TAFE. 

Institutional Evaluation Models for TAFE 

Institutional evaluation has been studied in the TAFE 
context by Byrne and others (Byrne, Thomson, 1985; Thomson, 
Byrne, Houston, 1984). These studies sought to develop 
'system-wide' evaluation models suited to the bureaucratic 
structure of TAFE in Australia which are designed to promote 
a means of improving performance. The models proposed 
promote a global approach to evaluation which is similar to 
models adopted in the US and other western countries for 
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individual institutional evaluation. in so doing, the 
authors have adopted the view of educational auditing as 
the methodical and systematic review of the whole 
situation associated with an institution*. 

In comparison with several models of evaluation of American 
colleges, in particular that in Portland, Oregon, Burne and 
his fellow researchers identify several features which are 
claimed to have direct relevance to TAFE. Among these 
features are the need to use a well qualified and 

experienced external consultant, recognition of the need for 
a flexible approach in recording the reality of a particular 
school ... the full and careful description of a school as a 
starting point for evaluation, recognition of the primacy of 
the Principal in the life of a school and the use of a 

case study approach to evaluation. 

xn developing models for TAFE, the authors refer to the need 
for a two stage evaluation - a profile phase and an inquiry 
phase. The need for an accurate college description is seen 
as an essential precursor to the evaluation. 

While the model is a valuable contribution to the study of 
evaluation in TAFE institutions and provides comprehensive 
data on evaluation instruments and methodology , the project 
does not give sufficient emphasis to evaluation as a 
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component of the on-going management structure. To do so 
would overcome many of the difficulties encountered, 
particularly those relating to time, cost and 
implementation . 

The development of College profiles however is seen as an 
important component in corporate planning but should be 
undertaken by the staff of the College or institution. 
Institutional evaluation may be required as part of the 
educational auditing process but it should not be seen as a 
substitute for ongoing (self) evaluation. 

Swedish Model for Self-Evaluation 

This concept of evaluation has been incorporated as part of 
the on-going management structure in a recent project in 
Sweden (described by Furamak, 1981). This project was 
designed to foster institutional self-study and 
self-evaluation and resulted from the decentralisation of 
the Swedish Higher Education system in 1977. This move 
closely parallels. in time and action, the move to 
decentralise administration throughout TAPE in NSW. In 
Sweden, as in NSW, this led to structural changes in the 
organisation, increased public involvement in the decision 
making process, and the delegation of authority and 
responsibility to colleges. In describing these changes in 



49 



Sweden, Furamak refers to the move towards a more 

dynamic process-oriented perspective where self-evaluation 
and the capacity for self-renewal are key concepts'. The 
Swedish concept of self-evaluation is directly related to 
'activity evaluation' and is seen as a 'continuous 
systematic activity' to be undertaken by each institution. 
The information resulting from self-evaluation is primarily 
intended for decision making purposes. It is therefore a 
'grass roots' approach to evaluation which ultimately 
involves all staff, 

Furamak states succinctly that the purpose of the project 
is to stimulate and foster critical self-study at all 
levels of the higher education system with the ultimate aim 
to improve effectiveness and efficiency'. 

The support of all staff for the concept of self-evaluation 
was regarded as central to the success of the project. 
Conferences, workshops and personal contacts were used to 
encourage and gain support for the management concept. 

The main features of the project design were: 

(i) establishing staff support for the purpose of 
self-evaluation ; 

(ii) securing staff concensus on project design; 
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(iii) identifying strengths and weaknesses of present 
activities and performance; 

(iv) the formulation of goals for change; 

(v) the adoption of an action plan for self-evaluation. 

The Swedish project stressed the importance of selecting the 
right issues for improvement , issues which ' . . . will 
contribute to innovation and growth within the institution'. 
To identify these isrues, the process of 'problem scanning' 
is seen as the initial step to self-evaluation. This is 
best achieved by comparing program impact with educational 
or institutional (corporate) goals. 

Self-evaluation as typified by the Swedish project 
recognises the need for on-going feedback in the educational 
management process . 

It encourages staff participation in decision making, the 
setting of goals, the use of profiles and action plans and 
the identification of areas or issues for improvement which 
enable the institution to meet the changing needs of 
society. Such a management process is dynamic in operation 
and provides for a more effective and efficient performance. 

While the main aim of self-evaluation in the Swedish context 
is to provide a better basis for future decision making, 
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concern was expressed over the possible misuse of 
information due to apathy or more deliberate intentions, 
Romney (1979) refers to problems of morale relating to 
institutional evaluation and to the expose of 'cherished 
activities and practices'. The need for good communication 
and appropriate staff development is strongly advocated to 
overcome such problems of staff morale. 

The Swedish project is significant in that it advocates for 
educational institutions a process of self-evaluation which 
forms an integral part of the management process. It 
indicates also the need to identify specific issues which 
require more detailed evaluation and from which changes to 
existing goals and operations may result. 

Thus it wil** be seen that institutional evaluation may have 
limited appeal compared with the ongoing self evaluation 
typified by the Swedish project. 

This model of evaluation is particularly relevant to 
developments in NSW TAPE c\nd these aspects will be examined 
in some detail in Part II of this paper. 
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PART II 



COMPREHENSIVE EVALUATION IN TAFE IN NSW 



Introduction 

The first part of this paper traced the developments leading 
to increased demand for accountability in education. It 
established the need for more evaluation of educational 
activities and outcomes to improve the decision making 
process and provide for increased demands for 
accountability. Superimposed upon this management process 
is the movement towards comprehensive auditing or review of 
the organisation's operations. 

In the second part of this paper the impact of these forces 
for change on TAF2 in NSW is examined and the organisational 
re-structuring and the management systems introduced to 
accommodate these developments are described and discussed • 

TAFE in NSW - Its Diversity and Scope 

The NSW Department of TAFE is a major provider of tertiary 
education to communities statewide. It achieves this 
through the work of over 100 colleges and 200 centres 
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throughout NSW. It provides over 1200 established cours s 
of study which have statewide accreditation, and a wide 
range of progirarus to meet the needs of disadvantaged groups 
in the community including those in rural areas. The 
diversity of NSW 'VAFE and the present demands upon its 
resources have meant rapid change in program content and 
educational direction. Many of these changes have been 
government initiatives to counter current economic 
difficulties; to respond to changes in demography and social 
values; to encourage technological development in industry 
and commerce; and to meet the needs of disadvantaged 
community groups. 

With the introduction of the TAFE Act in 1974, enrolments in 
NSW TAFE have more than doubled to over 400,000 with staff 
levels in excesr of 10,000. Government initiatives and 
changing community needs have required TAFE to develop a 
management structure which is flexible and responsive to the 
changing face of tertiary education. The move to review the 
management structure of the department was influenced by 
government action which led to the preparation of several 
'statements of purpose'. 

In 1974, the then NSW Department of Technical Education 
published a document entitled 'Technical Education - 
Preparing for Tomorrow'. This document was the department's 
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submission to the Australian Committee on Technical and 
Further Education which, in 1974, also released its report 
entitled 'TAPE in Australia - Report on Needs in Technical 
and Further Education*, It was the publication of this 
latter document which increased public and government 
awareness of the potential of TAFE and led to a fundamental 
shift in direction in its educational provision. 

In 1977, the department decentralised its administration by 
moving rapidly from a highly-centralised administrative 
structure to one based on regions. While certain 
administrative functions are still performed centrally, for 
example, the payment of staff salaries, there has been a 
significant shift towards the college as the operational 
unit of the department's activities. Since 1977, further 
administrative delegations to regions and colleges have 
occurred in an attempt to provide an appropriate balance of 
responsibilities best suited to ensure the most effectivf) 
and efficient use of resources and an appropriate degree of 
flexibility to meet community needs. 

Corporate Planning as a Manage^zcrc Tool 

Government initiatives to provide greater measures of 
accountability and more effective and efficient operation of 
government departments and agencies have been described 




earlier in this paper. In response to these governrut.nt 
initiatives and increasing public awareness and to provide 
direction to its educational administrators, NSW TAFE 
introduced the concept of corporate planning. The intention 
was to set '..o the bench-marks for the annual cycles of 
planning and operation against which achievements with 
respect to both short-term and long-term goals and 
priorities can be measured'. (The Corporate Plan for TAFE 
in NSWr 1S83) . 

The issuing of a corporate plan in 1983 was seen as the 
first stage of an improved corporate management process. 
The plan provided a broad statement of purpose and included 
a broad corporate goal together with several more specific 
strategic goals. 

Implementation of corporate planning was to be achieved by 
the preparation of strategic programs at all levels of 
educational management in the department. These programs 
incorporate acl:ion plans designed to achieve the 
depa^-tment 's goals and objectives. 

To monitor the degree success in the matching of 

operations to goals, a third phase of corporate planning is 
required. This is the evaluation phase. The evaluation is 
compared to the organisation goals and any disparity signals 
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the need to review either the goals, the strategies for 
their implementation or the programs being offered. 

This three-phase corporate planning model is consistent with 
miny of the organisational and administrative advantages of 
management by objectives discussed earlier. This planning 
cycle is represented diagrammatically in Figure 3. 

The introduction of the three-phase model for corporate 
planning has provided NSW TAPE with: 

(i) a better-structured basis for organisational decision 
making; 

(ii) increased participation by staff and other 
stakeholders in educational decision making , 
particularly in the strategic programming and 
self-evaluation phases; 

(iii) improved communication within the department with a 
more reliable data base for future decision making and 
for accountability requirements; 

(iv) the establishment of priorities and the development of 
clear guidelines for program development and 
implementation; 

(v) the means to identify issues for improvement which 




will contribute to innovation and change and more 
efficient and effective operation; 

(vi) the identification of realistic program goals and the 
monitoring of the impact of such programs; 

(vii) improved flexibility of operation and more effective 
and efficient use of educational resources. 



Strategic Planning 

Implementation of the corporate plan was followed by the 
development of strategic plans at all levels of management 
within the Department. Here the term 'strategic planning' 
refers to the means by which the goals and objectives which 
relate to an organisation's field of endeavour are converted 
to actions in the form of programs. This form of planning 
develops the strategies whereby these goals and objectives 
may be achieved. These strategic plans are usually prepared 
covering a three year period which is consistent with the 
department's triennial planning sut.^issions to the 
Commonwealth Tertiary Education Commission. 
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strategic planning for this purpose should be broad enough 
to provide an appropriate basis for action or operational 
planning. Thus, in NSW TAFE, the development of strategic 
plans were undertaken at three major levels of activity: 
departmental, by Colleges and Schools and by the servicing 
units. 

(a) Departmental Strategic Plans 

In NSW TAFE, departmental strategic plans are 
initiated at senior management level and establish 
strategies for planning throughout the department. To 
date strategic planning has included major areas of 
activities such as: 

... regionalisation ; 

student examination and assessment procedures; 
... library services; 

development of educational computing activities. 

These plans have been developed to be achievable 
within se;; time limits. This, however, is dependent 
upon budget constraints and for this reason strategic 
programs need to be reviewed annually. 

(b) College/School Strategic Plans 

In NSW, colleges have the role of being the 
operational units of the department. In order to 
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clearly identify this role College profiles are 
developed as a basis for strategic planning, NSW TAFE 
has traditionally based its educational quality 
assurance upon a system of teaching schools each with 
responsibilities in specific disciplines. The schools 
have state-wide responsibility for maintaining 
uniformity of an acceptable educational standard and 
provide a high level of educational consultancy to the 
colleges • 

The underlying factors for establishing strategic 
planning at the school and college levels are: 

(i) the stating of goals and objectives relevant to 
the particular activities of each college or 
school; 

(ii) the identification of major stakeholders and the 
establishing of appropriate lines of 
communication. This may be achieved by external 
representation on college or school committees 
or by direct personal contact with individuals 
or groups, or by structured surveys; 

(iii) the consideration of positive and negative 
educational growth factors. For this purpose 
the acquisition of data from within the 
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organisation and from external sources is 
required ; 

(iv) the determination of the education resource 
potential within a college or school. The range 
and scope of the educational provision and the 
extent to which the college or school can meet 
the future needs of stakeholders will depend 
upon the total resources available and the level 
of efficiency and effectiveness in the use of 
these resources. To achieve these objectives , 
the degree of flexibility in the utilisation of 
these *-esources is important in considering 
strategies for program implementation; 

(v) the development of future educational policy. 
Future policy at the school/college level is 
dependent upon overall departmental policy and 
should be based upon past performance and 
present needs. For this reason, the evaluation 
phase of corporate planning is a vital link in 
the total management system. From such 
evaluations of past performance, the strengths 
and weaknesses of an organisation can be 
identified and appropriate action taken; 
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(vi) the specification of strategies lor goal 
achievement. The strategic plan should clearly 
state the strategies by which the educational 
goals can be achieved. It is these strategies 
which also provide the basis for future 
monitoring and evaluation. 

In addition to the factors listed above, the teaching 
schools, because of their statewide responsibilities , 
give prime consideration to the maintenance of 
educational quality. In this regard, strategies are 
being developed to improve the efficiency, 
effectiveness and appropriateness of all educational 
programs. Such strategies include the need to develop 
more flexible use of resources and the introduction of 
a variety of teaching methodologies to achieve 
educational objectives. 

Servicing Units Strategic Plans 

The KS!V TAFE servicing units and divisions are 
predominantly centrally located and provided a full 
range of educational and administrative support to 

/-x^l "? X - /-I ^ ^ ^ ^. ^ V> ^ ^ "! — 

In developing strategic plans for these units of TAFE, 
government initiatives and the provision of special 
funding are important considerations. 



staff participation in Strategic Planning 

In the development of strategic plans at all levels of the 
organisation it is essential that staff be given the 
opportunity to participate. The working parties established 
for this purpose have been of a size consistent with the 
extent of the area of influence on departmental operations. 
An oversized committee, however while providing broader 
representation, can extend the period of planning to 
unacceptable limits . Where appropr iate , external 

stakeholders are represented on the working parties. 



Action or Operational Planning 

Strategic programs also provide the basis for operational or 
action planning. Action planning is the means by which the 
strategic plans are implemented. Action plans are usually 
developed for a shorter duration than strategic programs and 
are consistent with an operational cycle such as the 
academic year or semester. In some circumstances a shorter 
period may be appropriate. 

In developing action plans, managers in NSW TAPE have: 

(i) established goals and objectives appropriate to their 
field of operation. These shorter term goals are 
consistent with the department's corporate and 
strategic goals; 
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(ii) identified the range of clients for the educational 
programs to be provided; 

(iii) established the extent and nature of the educational 
resources available; 

(iv) determined priorities where resources may not suffice 
to meet the indicated nefjds of potential clients. In 
this regard, priority guidelines which have been 
incorporated into strategic plans are taken into 
consideration ; 

(v) allocated resources; 

(vi) planned operational activities including student 
enrolment, examination and assessment requirements and 
staff development. 

It should be emphasised that such action plans are: 

(i) based upon department/college/school strategic plans; 

(ii) developed with staff input and where possible student 
participation; 

(iii) documented to ensure full understanding by all staff 
of the educational directions for the coming period 
and to provide a basis for monitoring and 
self -evaluation • 



ERIC 



53 85 



The development of such plans at all levels of management 
within the department provides also the basis for operations 
evaluation . 



Evaluation in NSW TAFE 

Managerial effort in NSW TAFE is directed towards the 
provision of courses in Colleges to meet the needs of those 
in the community. Monitoring of the outcomes of the 
educational provisions occurs with employers and other 
stakeholders through the functioning of Regional Councils 
and college and School Committees. However, the present 
financial climate and requirements for increased 
accountability indicate a need for a more structured 
approach to evaluation. 

This evaluation process is an important component of 
educational management at all levels of TAFE in providing 
for these requirements. 

Monitoring 

Having implemented a range of programs as the result of 
strategic ana opex. ai. iOnaa. pisiinin-a ,- zn-. ^.„v.c_3 
monitoring ensures that the prograir.s are evaluated to 
establish their effectiveness, efficiency and appropriate- 
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ness. Monitoring is the critical observation of program 
operation on an ongoing basis. This usually refers to the 
day-to-day activities rf managers but can be expanded to 
incorporate activities related to ongoing program review by 



necessary feedback in determining the strengths and 
weaknesses of the programs provided. 

In NSW TAFE educational monitoring is based upon criteria 

used in planning and requires the use of indicators which 



indicators rather than comprehensive descriptions of 
performance reduces the time and effort involved in 
evaluation thus reducing the overall cost. The continued 
use of established validated indicators provides reliable 
information relating to trends. 

Such an evaluation process enables a manager to obtain data 
relating to: 

(i) program performance with regard to the inputs and 
outcomes ; 

{ii] the changing need for such programs; 

(iii) the leve] of appropriateness and effectiveness of the 
different strategies employed by the program; 



bodies external to local management , 



It provides the 



give some tangible measure of performance. 



The use of 
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(iv) the extent to which the program is meeting the 
corporate goals and strategies* 

Performance Indicators 

(a) Within NSW Government Departments ; 

The development of performance indicators for program 
evaluation by NSW government departments has been 
undertaken by Lenne & Wells (1986)- Wells defines an 
indicator as a variable (or composite of 

variables) which is used to represent a more complex 
concept, 'Performance* is a complex concept, and in 
the NSW public sector it is broken down into three 
components : appropriateness , effectiveness and 
efficiency'. These indicators are substitutes 
for things which cannot be measured directly' but 
which can, with careful interpretation, lead to 
judgements about the program, 

(b) Within NSW TAFE ; 

In TAFE educational outcomes are often qualitative and 
measures of program impact are difficult to quantify 
directly. Therefore the concept of using performance 
indicators for evaluation purposes Is considered 
appropriate. 
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In evaluating the offerings of a coT lege or regional 
TAPE program in NSW, Wells claims thct appropriateness 
is indicated by the degree to which the educational 
provision is matched with client needs. Program 
effectiveness then is indicated by the degree to which 
stated aims and objectives are achieved, while program 
efficiency is indicated by the extent of resource 
utilisation. 

The development of appropriate performance indicators 
is the basis for evaluation. The indicators selected 
are validated by relating them to the strategic 
planning goals developed during the second phase of 
corporate planning c 

The level of success of a performance indicator is 
dependent upor : 

(i) the degree of precision with which it can be 
measured ; 

(ii) the degree of representation of the particular 
aspect of performance being measured; 

(iii) the ^^.el of acceptability by stakeholders of 
the indicator as a measure of progiv^m 
effectiveness and efficiency. 
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As no single indicator can be expected to meet all 
such tests of validity, more than one indicator of 
performance is usually required. Since each indicator 
may measure a different aspect of performance, the use 
of a number of indicators enables a more complet . 
picture to be developed. The number of indicators 
selected depends upon the level of comprehensiveness 
required of the evaluation, 

Lenne & Wells identify three basic principles 
concerning the development and use of performance 
indicators. These are: 

(i) indicators are uevt ..\Ded by asking questions 
about certain elements of a program. These 
elements relate to needs , objectives , inputs , 
activities, outputs and impacts; 

(ii) indicators which are developed and implemented 
need to be reviewed and refined to ensure high 
levels of validity and reliability; 

(iii) indicators which are selected for use should be 
developed with the user in mind. In NSW TAFE 
these relate to managers and those stakeholders 
to whom the individual manager is accountable. 



ERIC 



70 

58 



(c) Indicator Interpretation ; 

The interpretation of information obtained by the 
application of indicators is particularly important 
and it is essential that agreement is reached among 
stakeholders on indicator interpretation as varying 
interpretations can lead to misunderstandings, 
confrontation and conflict. 

The question of possible misuse of performance 
indicators is raised by Roe & Moses in their report on 
departmental reviews of higher education institutions. 
They note that apart from the risk of failing to 

catc>. the essence of a department's performance there 
is the likelihood that their active presence would 
more and more determine that performance'. They 
agree , however , that performance indicators need not 
necessar ily be misused and that they can provide a 
means of regular self-evaluat ion and ' systematic and 
coherent information for departmental reviews ' . 

In NSW TAFE the development and use of performance 
indicators is proceeding with due care to ensure their 
validity and reliability. Testing of many indicators 
is required to establish their value to the total 
eval'iation process. 
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Evaluation Procedure for Program Monitoring 

The procedure for ongoing internal evaluation by managers in 
NSW TAPE, includes the following: 

(i) the determination of appropriate indicators of 
performance to ensure a degree of comprehensiveness of 
information required for planning and accountability; 

(iij tne seeking of agreement with stakeholder.^ of 
appropriateness of indicators and consensus on 
interpretation; 

(iii) the identification of indicators which will be used 
for accountability purposes. These may be required to 
provide data to higher management levels and to meet 
government requirements such as those required in the 
NSW Annual Reports Act (1985) ; 

(iv) the application o£ indicators to performance and the 
establishment of an information data base; 

(V) the interpretation of data to identify program 
strengths and weaknesses. Areas of weakness so 
identified may require further evaluation. 

Such a process of self-evaluation through monitoring is 
essential to the successful operation of the total 
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management process. Its success is dependent upon the 
agreed development of appropriate and valid performance 
indicators • It is not however- intended that the monitoring 
process will entail a substantial administrative increase in 
the management process as many of these activities in 
checking performance and assessing output are already in 
existence • 

Comprehensive Audit and Program Evaluation 

In 1984f Cooper and Craig developed a profile of internal 
audit in Australia. They recorded strong support from 
management for internal auditing and noted that the process 
presently covered a wide range of activities within 
organisations • It was predicted that in the future, the 
role of ill -^--nal auditing would expand to include the 
review of efficient operation and review of the 
effectiveness of management*. To accommodate this 

expansion, existing internal audit units will be 
strengthened to provide for program performance evaluation. 

Program Evaluation Activities in NSW TAFE 

In May 1986; a KiSW government requirement was issued to all 
ministers for the submission of Five Year Plans for Program 
Performance Review . These planj are be subir^itted to the 
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Management Performance Committee in December of each year 
for approval. 

For th3 purposes of performance evaluation, programs are 
defined as any set of procedures, activities, resources 

and/or management strategies designed to achieve some common 
goal(s) or ob ject ive (s) ' • The aims of the evaluation cycle 
are to: 

(i) assist managers to improve their operations; 

(ii) satisfy the accountability expectations of government, 
the community and clients; 

(iii) provide information for senior management concerning 
the appropriateness, effectiveness and efficiency of 
TAFE provision. 

It is expected that over a five year cycle, NSW TAFE's major 
objectives and activities will be reviewed. This 
comprehensive auditing process is to be a normal part of the 
department's review procedures. It is intended that these 
reviews will focus attention upon areas which have been 
identified by the monitoring process as those requiring more 
r igorous assessment . 
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Establishment of Directorate of Audit and Review 

Such a requirement will lead to the development of universal 
indicators of performance for use throughout the department 
and establish a structure for performance review. This is 
consistent with the aims of the monitoring process 
established as part of corporate planning. In fact, the 
program evaluation plan is designed to review in greater 
depth those aspects of performance which have been 
identified as requiring closer and more thorough attention. 
In so doing, use will be made of existing evaluation 
mechanisms within the department such as those currently 
established to review curricula and teacher assessment. 
Where such review processes are not in existence, program 
evaluation will be conducted on a project basis, by the 
creation of a task force appointed for the purpose, NSW 
TAPE is in the process of establishing a directorate of 
Audit and Review which will incorporate the existing 
Internal Audit Unit, (Figure 4), These project taskforces 
will comprise officers from the divisions within the 
directorate. 
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The Audit and Review Directorate of the Department of TAPE 
will embrace four major activities: 

(i) Internal audit which is the independent appraisal 
activity within the department providing a service to 
senior executi ves by assessing financial compliance 
systems and controls and alerting the department to 
weaknesses or undesirable departures from internal 
control procedures. 

(ii) Economy and efficiency reviews which are those 
activities within the department which determine 
whether resources are being managed in an economical 
and efficient manner. These reviews aim to reveal the 
causes of inefficiency or uneconomical practices, 
including inadequacies in management information 
systems or administrative procedures. 

(iii) Effectiveness reviews which are used to determine 
whether the objectives established by government and 
TAPE are being met and which examine alternatives 
which might yield the desired results more 
economically or efiectively. 

(iv) Computer systems audits which are the independent 
appraisals of the operations of the department's 
complex computer systems. Positive audit feedback 
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provides reliance on information generated by these 
computer-based information systems. 

In addition to the above functions, the Management Review 
Division of the Directorate of Audit and Review will be 
responsible for the development and implementation of the 
Five Year Program Evaluation Plan and for reviewing the 
reliability and validity of the performance indicators 
published by the Department. 

Program Performance Evaluation Plan 

On receiving notice from government of the requirements of 
the initial Five Year Program Evaluation Plan, the 
department sought advice from TAFE managers at Head of 
Division level and above of their intentions with regard to 
evaluating and reviewing their operations. These plans were 
received by the Comprehensive Audit Committee and seven 
major departmental activities were identified for review 
during the first year of the plan. The other review 
proposals were noted by the Committee and progress will be 
monitored by the Audit and Review Directorate. Many of 
these review proposals will use the department's existing 
mechanisms for evaluation and review. 

Figure 5 illustrates the range and exteat of these review 
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projects over the departments organisational structure. 
Level 1 projects involve all levels of management and are 
broader in concept than projects undertaken at lower 
management levels. These level 1 projects will be 
undertaken by the Management Review Division and fulfils the 
first year of the requirements of the departments rolling 
Five Year Program Evaluation Plan. 

Review projects identified at lower management levels are 
more specific in scope. The projects will be co-ordinated 
by the Management Review Division but will be undertaken by 
the management units concerned. The projects identify those 
aspects of program performance requiring a more detailed 
evaluation than would be possible during the normal 
monitoring process . 

Planning and Evaluation Support to Managers 

In order to assist NSW TAPE managers in the move towards the 
departments initiatives in corporate planning and program 
evaluation, a system of management support has been 
introduced to aid in developing thei r strategic and 
operational plans and in developing appropriate performance 
indicators v nd procedures for monitoring and evaluating 
their programs. 
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PROGRAM PERFORMANCE EVALUATION 
DISTRIBUTION OF FIVE YEAR PLAN 
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Figure 5 



A Pi ^nning and Evaluation Support Unit (PESU) has been 
established in NSW TAFE for this purpose. PESU will be 
responsible for establishing guidelines for each management 
level on: 

(i) the contents of, and methods for, developing 
objectives and strategic and operational plans; 

(ii) review of such plans; 

(iii) the methods of monitoring operation and the evaluation 
of selected programs. 

In addition, PESU have developed in co-operation with the 
Staff Development Division training packages to prepare 
managers for the requirements of corporate planning, program 
evaluation and tha development of performance indicators, 
and the revision at appropriate inte^ 'als of ^he 
department's corporate plan. 

Future Directions 

The model adopted by N3W TAFE for the planning, 
implementation and evaluation of its educational provision 
is designed to meet changes in community needs for ITAFE and 
the increasing demand for accountability. The successful 
operation of this management process must be the 
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responsibility of all managers in TAFE. The challenge of 
the future will be to provide educational programs of high 
quality wi'-hin a climate of financial constraints. To meet 
this challenge close monitoring of the efficiency, 
effectiveness and appropriateness of these programs will be 
a requirement of all managers. It will be their 
responsibility also to ensure that teaching methodology is 
not only of a high standard but that all available 
technology is used in appropriate circums .nces. This will 
require much co-ordinated experimentation. 

In addition future decision making must be based on the best 
available data and for this purpose efficient and effective 
management information systems are essential. The data upon 
which future decisions are based must be accurate and 
pertinent to the decision making process. For 
accountability purposes the data must be valid and ba.;ed on 
performance indicators which have proven reliability and 
general acceptance by stakeholders. All published data must 
be analysed and reported in a format which provides the 
rationale for decisions taken. Where problem areas are 
identified the management system must be able to react to 
take appropriate action. Once in place the procco.- 
described in this paper will enable NSW TAFE to achieve its 
corporate and strategic goals end to meet the challenges of 
the future with confidence. 
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